{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f5c95f47",
   "metadata": {},
   "source": [
    "# SSD样例开发\n",
    "\n",
    "[TOC]\n",
    "\n",
    "---\n",
    "\n",
    "## 1. MindStudio安装\n",
    "\n",
    "### 1.1 下载软件包\n",
    "\n",
    "cann 安装流程及软件包获取参看https://www.hiascend.com/document/detail/zh/canncommercial/51RC1/envdeployment/instg \n",
    "MindStudio软件包获取链接https://www.hiascend.com/software/mindstudio/download\n",
    "\n",
    "### 1.2 安装依赖\n",
    "\n",
    "检查系统是否安装python依赖以及gcc等软件。\n",
    "分别使用如下命令检查是否安装gcc，make以及python依赖软件等。\n",
    "```\n",
    "gcc --version\n",
    "g++ --version\n",
    "make --version\n",
    "cmake --version\n",
    "dpkg -l zlib1g-dev| grep zlib1g-dev| grep ii\n",
    "dpkg -l libbz2-dev| grep libbz2-dev| grep ii\n",
    "dpkg -l libsqlite3-dev| grep libsqlite3-dev| grep ii\n",
    "dpkg -l libssl-dev| grep libssl-dev| grep ii\n",
    "dpkg -l libffi-dev| grep libffi-dev| grep ii\n",
    "dpkg -l unzip| grep unzip| grep ii\n",
    "dpkg -l pciutils| grep pciutils| grep ii\n",
    "dpkg -l net-tools| grep net-tools| grep ii\n",
    "dpkg -l libblas-dev| grep libblas-dev| grep ii\n",
    "dpkg -l gfortran| grep gfortran| grep ii\n",
    "dpkg -l libblas3| grep libblas3| grep ii\n",
    "dpkg -l liblapack-dev| grep liblapack-dev| grep ii\n",
    "dpkg -l openssh-server| grep openssh-server| grep ii\n",
    "dpkg -l xterm| grep xterm| grep ii\n",
    "dpkg -l firefox| grep firefox| grep ii\n",
    "dpkg -l xdg-utils| grep xdg-utils| grep ii\n",
    "dpkg -l libdbus-glib-1-dev | grep libdbus-glib-1-dev  | grep ii\n",
    "dpkg -l gdb | grep gdb  | grep ii\n",
    "```\n",
    "\n",
    "若分别返回如下信息则说明已经安装，进入下一步（以下回显仅为示例，请以实际情况为准）。\n",
    "```\n",
    "gcc (Ubuntu 7.3.0-3ubuntu1~18.04) 7.3.0\n",
    "g++ (Ubuntu 7.3.0-3ubuntu1~18.04) 7.3.0\n",
    "GNU Make 4.1\n",
    "cmake version 3.10.2\n",
    "zlib1g-dev:arm64 1:1.2.11.dfsg-0ubuntu2 arm64        compression library - development\n",
    "libbz2-dev:arm64 1.0.6-8.1ubuntu0.2 arm64        high-quality block-sorting file compressor library - development\n",
    "libsqlite3-dev:arm64 3.22.0-1ubuntu0.3 arm64        SQLite 3 development files\n",
    "libssl-dev:arm64 1.1.1-1ubuntu2.1~18.04.6 arm64     Secure Sockets Layer toolkit - development files\n",
    "libffi-dev:arm64 3.2.1-8      arm64        Foreign Function Interface library (development files)\n",
    "unzip          6.0-21ubuntu1 arm64        De-archiver for .zip files\n",
    "pciutils       1:3.5.2-1ubuntu1 arm64        Linux PCI Utilities\n",
    "net-tools      1.60+git20161116.90da8a0-1ubuntu1 arm64        NET-3 networking toolkit\n",
    "libblas-dev:arm64 3.7.1-4ubuntu1 arm64        Basic Linear Algebra Subroutines 3, static library\n",
    "gfortran       4:7.4.0-1ubuntu2.3 arm64        GNU Fortran 95 compiler\n",
    "libblas3:arm64 3.7.1-4ubuntu1 arm64     Basic Linear Algebra Reference implementations, shared library\n",
    "liblapack-dev:arm64 3.7.1-4ubuntu1 arm64        Library of linear algebra routines 3 - static version\n",
    "openssh-server 1:7.6p1-4ubuntu0.5 arm64        secure shell (SSH) server, for secure access from remote machines\n",
    "xterm          330-1ubuntu2 arm64        X terminal emulator\n",
    "firefox        83.0+build2-0ubuntu0.18.04.2 arm64        Safe and easy web browser from Mozilla\n",
    "xdg-utils      1.1.2-1ubuntu2.5 all          desktop integration utilities from freedesktop.org\n",
    "ii  libdbus-glib-1-dev 0.110-2      arm64        deprecated library for D-Bus IPC (development files)\n",
    "ii  gdb            8.1.1-0ubuntu1 arm64        GNU Debugger\n",
    "```\n",
    "否则请执行如下安装命令（如果只有部分软件未安装，则如下命令修改为只安装还未安装的软件即可）："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38dbec57",
   "metadata": {},
   "outputs": [],
   "source": [
    "!apt-get install -y gcc g++ make cmake zlib1g-dev libbz2-dev libsqlite3-dev libssl-dev libffi-dev unzip pciutils net-tools libblas-dev gfortran libblas3 liblapack-dev openssh-server xterm firefox xdg-utils libdbus-glib-1-dev gdb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfb222c6",
   "metadata": {},
   "source": [
    "### 1.3 安装MindStudio\n",
    "\n",
    "解压MindStudio_{version}_linux.tar.gz软件包:\n",
    "```\n",
    "tar -zxvf MindStudio_{version}_linux.tar.gz\n",
    "```\n",
    "解压jbr至MindStudio安装根目录，jbr下载链接请根据操作系统架构选择。\n",
    "\n",
    "x86_64链接：https://cache-redirector.jetbrains.com/intellij-jbr/jbr_dcevm-11_0_10-linux-x64-b1341.35.tar.gz  \n",
    "aarch64链接：https://cache-redirector.jetbrains.com/intellij-jbr/jbr-11_0_10-linux-aarch64-b1341.35.tar.gz  \n",
    "解压jbr至MindStudio根目录后目录结构如下：\n",
    "```\n",
    "├── bin\n",
    "├── jbr        //确认将压缩包中的jbr文件夹解压至MindStudio根目录           \n",
    "├── lib                              \n",
    "├── ......  \n",
    "```\n",
    "使用MindStudio的安装用户进入软件包解压后的MindStudio/bin目录，执行如下命令：\n",
    "```\n",
    "cd MindStudio/bin\n",
    "./MindStudio.sh\n",
    "```\n",
    "\n",
    "**备注：MindStudio安装流程详细请参照https://support.huaweicloud.com/devg-mindstudio304/atlasms_02_0002.html**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dae9da8",
   "metadata": {},
   "source": [
    "## 2. 样例开发流程图\n",
    "\n",
    "![典型样例开发](./流程.PNG)\n",
    "\n",
    "## 3. 环境配置\n",
    "\n",
    "### 3.1 安装依赖包\n",
    "\n",
    "- python==3.7.5\n",
    "- cann==5.1.RC1\n",
    "- mindspore==1.7.0 (Ascend)\n",
    "- noah_vega==1.8.1 (AutoML工具)\n",
    "\n",
    "cann安装版本需根据自身服务器类型选择`x86_64`或者`aarch64`,使用以下命令可查看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d96a22d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "!arch "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53759f28",
   "metadata": {},
   "source": [
    "# 3.2 环境变量配置\n",
    "\n",
    "本段编辑环境变量文件过程均在**命令行**中完成，请打开并执行`cd /{文件路径}/workspace/`进到目录下\n",
    "\n",
    "#### 环境变量配置\n",
    "\n",
    "为防止多开发者使用同一服务器，环境搭建存在冲突，当前全部环境变量配置在shell脚本中，下面先创建一个新脚本文件：\n",
    "\n",
    "```\n",
    "vim env.sh\n",
    "```\n",
    "\n",
    "#### env.sh脚本内环境变量配置\n",
    "\n",
    "现在进入到`env.sh`的编辑界面，按`i`，界面最下方出现`-- INSERT --`表示进入**编辑模式**\n",
    "\n",
    "1. 根据自己程序的所使用的python环境配置PYTHONPATH:\n",
    "```\n",
    "export PYTHONPATH=/{文件路径}/lib/python3.7/site-packages/:$PYTHONPATH\n",
    "```\n",
    "2. 驱动与cann安装后默认会在`/usr/local/Ascend`目录下生成自己的环境变量配置脚本（如cann安装在自定义目录，请配置自定义目录），`source`以下两个脚本:\n",
    "```\n",
    "source /usr/local/Ascend/driver/bin/setenv.bash\n",
    "source /usr/local/Ascend/ascend-toolkit/set_env.sh\n",
    "```\n",
    "3. 配置单卡单模型，多卡多模型并行环境变量，`/ascend-toolkit`目录下可选取安装好的cann版本文件夹，如本样例的`5.1.RC1`，也可以是软链接到`5.1.RC1`的`latest`，如下:\n",
    "```\n",
    "export install_path=/usr/local/Ascend/ascend-toolkit/5.1.RC1\n",
    "```\n",
    "```\n",
    "export LD_LIBRARY_PATH=${install_path}/compiler/lib64/:$LD_LIBRARY_PATH\n",
    "export TBE_IMPL_PATH=${install_path}/opp/op_impl/built-in/ai_core/tbe:$TBE_IMPL_PATH\n",
    "```\n",
    "```\n",
    "export JOB_ID=10087  #该项不建议改动\n",
    "export DEVICE_ID=2  #单卡训练使用的device_id\n",
    "```\n",
    "```\n",
    "export NPU_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \n",
    "```\n",
    "配置完成，按`Esc`按键，下方`-- INSERT --`消失，表示退出**编辑模式**，按`:wq`然后按`Enter`结束文件编辑。\n",
    "\n",
    "#### 执行环境变量脚本\n",
    "```\n",
    "source env.sh\n",
    "```\n",
    "#### QA\n",
    "\n",
    "如果后续执行代码运行出现如下错误时，添加环境变量LD_PRELOAD\n",
    "\n",
    "```python\n",
    "ImportError: /root/.../lib/python3.7/site-packages/sklearn/__check_build/../../scikit_learn.libs/libgomp-d22c30c5.so.1.0.0: cannot allocate memory in static TLS \n",
    "```\n",
    "\n",
    "环境变量增加项（**路径根据自己实际出错的路劲做修改**）:\n",
    "```\n",
    "export LD_PRELOAD=/root/.../lib/python3.7/site-packages/sklearn/__check_build/../../scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd7f5968",
   "metadata": {},
   "source": [
    "## 4. 模型自动量化调优\n",
    "\n",
    "模型自动量化调优的基本运行过程\n",
    "\n",
    "**量化阶段（nas）**  \n",
    "    步骤1 采样量化模型.  \n",
    "    步骤2 微调训练量化后模型，评估模型精度。  \n",
    "    步骤3 将量化后的模型发送到评估服务器上，评估latency。   \n",
    "    重复步骤1、2、3，经过N次采样和学习，量化结束，选取若干个top精度的量化模型作为输出结果。  \n",
    "\n",
    "  **备注**：量化阶段的每次采样过程需要使用之前采样再训练的反馈结果作为输入，从而计算出调整后的采样策略，因此采样->训练->再采样->再训练的流程是线性的，无法使用多卡并行训练来加速采样。\n",
    "\n",
    "\n",
    "\n",
    "### 4.1 模型及数据集准备\n",
    "\n",
    "- **模型**：mindspore的ssd模型，模型来源于mindspore的modelzoo。\n",
    "用户需要对下载的模型脚本做一定的修改，从而满足自动量化调优的需求，主要包括以下几个方面的改动：  \n",
    "1）添加get_model方法：主要是为了提供一个接口给模型自动量化调优来获取脚本中定义的模型。  \n",
    "2）添加calib_func函数，该函数是模型前向训练过程的简化版本，只需要进行前向计算即可，用于量化后的矫正。  \n",
    "3）修改eval_net函数，主要是修改该函数的入参，将量化后的模型传递给该函数，而不是使用原始定义的模型。  \n",
    "具体的修改方法可以参考这里的readme文档：`{CANN包安装路径}/ascend-toolkit/latest/tools/ascend_automl/examples/detection/mindspore/quant/ssd_resnet50_fpn.md`\n",
    "\n",
    "- **模型权重和数据集**：使用coco数据集训练得到的模型权重文件。\n",
    "\n",
    "### 4.2 模型调优文件配置\n",
    "\n",
    "模型调优文件配置主要参照对应的样例目录`{CANN包安装路径}/ascend-toolkit/latest/tools/ascend_automl/examples/`，以ssd举例，配置文件参照目录如下：  \n",
    "`{CANN包安装路径}/ascend-toolkit/latest/tools/ascend_automl/examples/detection/mindspore/quant/ssd_resnet50_fpn.yml`\n",
    "\n",
    "当前样例已给出`./ssd_resnet50_fpn.yml`，配置介绍[分段模型调优配置介绍](#5.分段模型调优配置介绍)\n",
    "\n",
    "### 4.3 执行AutoML工具\n",
    "\n",
    "执行如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "853bb161",
   "metadata": {},
   "outputs": [],
   "source": [
    "!vega ssd_resnet50_fpn.yml -d NPU"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b66f9cf",
   "metadata": {},
   "source": [
    "### 4.4 AutoML输出展示\n",
    "  \n",
    "1. **量化阶段（nas）**\n",
    "\n",
    "   nas部分抽取了20个样本模型中的三个模型（第一个为没有量化的原始模型），每个模型的运行情况如下：\n",
    "\n",
    "   ```sh\n",
    "2022-03-17 16:32:41,614 INFO submit trainer, id=1\n",
    "2022-03-17 16:32:49,242 INFO Model was created.\n",
    "2022-03-17 16:32:49,242 INFO load model weights from file, weights file=/xxxxxx/models/ssd_resnet50_fpn_ascend_v130_coco2017_official_cv_bs64_acc37.56.ckpt\n",
    "2022-03-17 16:32:51,333 INFO current code: [32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32]\n",
    "2022-03-17 16:32:51,334 INFO quant_layer_index_list: [7, 12, 17, 22, 97, 102, 107, 112, 184, 186, 195, 197, 199, 203, 206, 208, 210, 214, 216, 218, 223, 225, 227, 231, 234, 236, 238, 242, 244, 246, 250, 252, 254, 259, 261, 263, 267, 270, 272, 274, 278, 280, 282, 286, 288, 290, 294, 296, 298, 302, 304, 306, 311, 313, 315, 319, 322, 324, 326, 330, 332, 334, 339, 340, 341, 344, 348, 352, 358, 362]\n",
    "2022-03-17 16:33:56,620 INFO start to calibrate model...\n",
    "2022-03-17 16:33:58,375 INFO calibrate model dataset size is: 36.64\n",
    "2022-03-17 16:35:17,319 INFO calib is done!\n",
    "2022-03-17 16:35:29,481 INFO compress_ratio:0.000%,flops:38.365G,params:132985.336K\n",
    "2022-03-17 16:35:29,491 INFO Start to eval accuracy...\n",
    "2022-03-17 16:44:55,222 INFO Finished eval accuracy, result is {'mAP': 0.3755167004355987}\n",
    "2022-03-17 16:44:55,230 INFO accuracy:0.3755167004355987\n",
    "2022-03-17 16:44:56,284 INFO Start to export air file: /xxxxxx/automl/tasks/quant_ssd_317/workers/nas/1/model_1.air\n",
    "2022-03-17 16:45:04,104 INFO Export air model successfully.\n",
    "2022-03-17 16:45:17,432 INFO evalaute model without loading weights file\n",
    "2022-03-17 16:45:20,358 INFO Model was created.\n",
    "2022-03-17 16:45:20,359 INFO The job id of evaluate service is nas_1_20220317164520359724.\n",
    "2022-03-17 16:46:28,713 INFO Evaluate sucess! The latency is 74.628.\n",
    "2022-03-17 16:46:28,714 INFO The latency in Davinci is 74.628 ms.\n",
    "2022-03-17 16:46:28,715 INFO valid performance: {'latency': 74.628}\n",
    "2022-03-17 16:46:28,724 INFO finished device evaluation, id: 1, performance: {'latency': 74.628}\n",
    "2022-03-17 16:46:31,586 INFO Update Success. step_name=nas, worker_id=1\n",
    "2022-03-17 16:46:31,587 INFO Best values: [{'worker_id': '1', 'performance': {'compress_ratio': 0.0, 'flops': 38.36510432, 'params': 132985.336, 'accuracy': 0.3755167004355987, 'latency': 74.628}}]\n",
    "   ```\n",
    "\n",
    "   ```sh\n",
    "2022-03-17 16:46:31,962 INFO => Final action list: [8, 32, 8, 8, 32, 32, 32, 32, 32, 8, 8, 32, 32, 32, 8, 8, 8, 32, 32, 8, 32, 8, 32, 32, 8, 8, 32, 32, 32, 8, 32, 8, 8, 32, 8, 8, 8, 32, 32, 8, 8, 32, 8, 8, 8, 32, 32, 8, 32, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]\n",
    "2022-03-17 16:46:31,974 INFO submit trainer, id=2\n",
    "2022-03-17 16:46:39,084 INFO Model was created.\n",
    "2022-03-17 16:46:39,085 INFO load model weights from file, weights file=/xxxxxx/models/ssd_resnet50_fpn_ascend_v130_coco2017_official_cv_bs64_acc37.56.ckpt\n",
    "2022-03-17 16:46:41,168 INFO current code: [8, 32, 8, 8, 32, 32, 32, 32, 32, 8, 8, 32, 32, 32, 8, 8, 8, 32, 32, 8, 32, 8, 32, 32, 8, 8, 32, 32, 32, 8, 32, 8, 8, 32, 8, 8, 8, 32, 32, 8, 8, 32, 8, 8, 8, 32, 32, 8, 32, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]\n",
    "2022-03-17 16:46:41,169 INFO quant_layer_index_list: [7, 12, 17, 22, 97, 102, 107, 112, 184, 186, 195, 197, 199, 203, 206, 208, 210, 214, 216, 218, 223, 225, 227, 231, 234, 236, 238, 242, 244, 246, 250, 252, 254, 259, 261, 263, 267, 270, 272, 274, 278, 280, 282, 286, 288, 290, 294, 296, 298, 302, 304, 306, 311, 313, 315, 319, 322, 324, 326, 330, 332, 334, 339, 340, 341, 344, 348, 352, 358, 362]\n",
    "2022-03-17 16:47:51,145 INFO start to calibrate model...\n",
    "2022-03-17 16:47:53,656 INFO calibrate model dataset size is: 36.64\n",
    "2022-03-17 16:52:17,900 INFO calib is done!\n",
    "2022-03-17 16:52:31,765 INFO compress_ratio:60.628%,flops:38.365G,params:52358.398K\n",
    "2022-03-17 16:52:31,774 INFO Start to eval accuracy...\n",
    "2022-03-17 17:02:06,005 INFO Finished eval accuracy, result is {'mAP': 0.3755730211849307}\n",
    "2022-03-17 17:02:06,019 INFO accuracy:0.3755730211849307\n",
    "2022-03-17 17:02:08,343 INFO Start to export air file: /xxxxxx/automl/tasks/quant_ssd_317/workers/nas/2/model_2.air\n",
    "2022-03-17 17:02:18,034 INFO Export air model successfully.\n",
    "2022-03-17 15:23:38,263 INFO evalaute model without loading weights file\n",
    "2022-03-17 15:23:40,944 INFO Model was created.\n",
    "2022-03-17 15:23:40,945 INFO The job id of evaluate service is nas_2_20220317152340945120.\n",
    "2022-03-17 15:25:04,270 INFO Evaluate sucess! The latency is 67.09.\n",
    "2022-03-17 15:25:04,296 INFO The latency in Davinci is 67.09 ms.\n",
    "2022-03-17 15:25:04,296 INFO valid performance: {'latency': 67.09}\n",
    "2022-03-17 15:25:04,306 INFO finished device evaluation, id: 2, performance: {'latency': 67.09}\n",
    "2022-03-17 17:02:31,409 INFO evalaute model without loading weights file\n",
    "2022-03-17 17:02:34,113 INFO Model was created.\n",
    "2022-03-17 17:02:34,114 INFO The job id of evaluate service is nas_2_20220317170234114646.\n",
    "2022-03-17 17:03:56,782 INFO Evaluate sucess! The latency is 66.939.\n",
    "2022-03-17 17:03:56,783 INFO The latency in Davinci is 66.939 ms.\n",
    "2022-03-17 17:03:56,783 INFO valid performance: {'latency': 66.939}\n",
    "2022-03-17 17:03:56,792 INFO finished device evaluation, id: 2, performance: {'latency': 66.939}\n",
    "2022-03-17 17:03:59,428 INFO Update Success. step_name=nas, worker_id=2\n",
    "2022-03-17 17:03:59,430 INFO Best values: [{'worker_id': '2', 'performance': {'compress_ratio': 60.62844252241465, 'flops': 38.36510432, 'params': 52358.398, 'accuracy': 0.3755730211849307, 'latency': 66.939}}]\n",
    "   ```\n",
    "\n",
    "   ```sh\n",
    "2022-03-17 21:17:14,879 INFO => Final action list: [8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 32, 8, 8, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 32, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 32, 32, 8, 32, 32, 8, 8, 8, 32, 32, 8, 8, 8, 32, 8, 8, 8, 8, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]\n",
    "2022-03-17 21:17:14,892 INFO submit trainer, id=14\n",
    "2022-03-17 21:17:22,118 INFO Model was created.\n",
    "2022-03-17 21:17:22,120 INFO load model weights from file, weights file=/xxxxxx/models/ssd_resnet50_fpn_ascend_v130_coco2017_official_cv_bs64_acc37.56.ckpt\n",
    "2022-03-17 21:17:24,305 INFO current code: [8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 32, 8, 8, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 32, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 32, 32, 8, 32, 32, 8, 8, 8, 32, 32, 8, 8, 8, 32, 8, 8, 8, 8, 8, 32, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8]\n",
    "2022-03-17 21:17:24,307 INFO quant_layer_index_list: [7, 12, 17, 22, 97, 102, 107, 112, 184, 186, 195, 197, 199, 203, 206, 208, 210, 214, 216, 218, 223, 225, 227, 231, 234, 236, 238, 242, 244, 246, 250, 252, 254, 259, 261, 263, 267, 270, 272, 274, 278, 280, 282, 286, 288, 290, 294, 296, 298, 302, 304, 306, 311, 313, 315, 319, 322, 324, 326, 330, 332, 334, 339, 340, 341, 344, 348, 352, 358, 362]\n",
    "2022-03-17 21:22:45,127 INFO start to calibrate model...\n",
    "2022-03-17 21:22:47,256 INFO calibrate model dataset size is: 36.64\n",
    "2022-03-17 21:29:21,452 INFO calib is done!\n",
    "2022-03-17 21:31:40,967 INFO compress_ratio:65.638%,flops:38.365G,params:45696.070K\n",
    "2022-03-17 21:31:40,979 INFO Start to eval accuracy...\n",
    "2022-03-17 21:41:51,712 INFO Finished eval accuracy, result is {'mAP': 0.37601061732166347}\n",
    "2022-03-17 21:41:51,720 INFO accuracy:0.37601061732166347\n",
    "2022-03-17 21:41:53,959 INFO Start to export air file: /xxxxxx/automl/tasks/quant_ssd_317/workers/nas/14/model_14.air\n",
    "2022-03-17 21:42:02,538 INFO Export air model successfully.\n",
    "2022-03-17 21:42:16,281 INFO evalaute model without loading weights file\n",
    "2022-03-17 21:42:18,954 INFO Model was created.\n",
    "2022-03-17 21:42:18,957 INFO The job id of evaluate service is nas_14_20220317214218957241.\n",
    "2022-03-17 21:43:42,353 INFO Evaluate sucess! The latency is 62.595.\n",
    "2022-03-17 21:43:42,355 INFO The latency in Davinci is 62.595 ms.\n",
    "2022-03-17 21:43:42,355 INFO valid performance: {'latency': 62.595}\n",
    "2022-03-17 21:43:42,367 INFO finished device evaluation, id: 14, performance: {'latency': 62.595}\n",
    "2022-03-17 21:43:45,378 INFO Update Success. step_name=nas, worker_id=14\n",
    "2022-03-17 21:43:45,385 INFO Best values: [{'worker_id': '14', 'performance': {'compress_ratio': 65.63826405642197, 'flops': 38.36510432, 'params': 45696.07, 'accuracy': 0.37601061732166347, 'latency': 62.595}}]\n",
    "   ```\n",
    "\n",
    "automl整体时间消耗，本次测试nas部分为单卡运行\n",
    "\n",
    "```sh\n",
    "2022-03-18 00:19:21,021 INFO ------------------------------------------------\n",
    "2022-03-18 00:19:21,021 INFO   Pipeline end.\n",
    "2022-03-18 00:19:21,021 INFO\n",
    "2022-03-18 00:19:21,021 INFO   task id: quant_ssd_317\n",
    "2022-03-18 00:19:21,022 INFO   output folder: /xxxxxx/automl/tasks/quant_ssd_317/output\n",
    "2022-03-18 00:19:21,022 INFO\n",
    "2022-03-18 00:19:21,022 INFO   running time:\n",
    "2022-03-18 00:19:21,022 INFO                nas:  7:47:22  [2022-03-17 16:31:43.358406 - 2022-03-18 00:19:05.934051]\n",
    "2022-03-18 00:19:21,022 INFO\n",
    "2022-03-18 00:19:21,029 INFO   result:\n",
    "2022-03-18 00:19:21,031 INFO    14:  {'compress_ratio': 65.63826405642197, 'flops': 38.36510432, 'params': 45696.07, 'accuracy': 0.37601061732166347, 'latency': 62.595}\n",
    "2022-03-18 00:19:21,031 INFO    16:  {'compress_ratio': 68.93762933380866, 'flops': 38.36510432, 'params': 41308.398, 'accuracy': 0.37569356691431266, 'latency': 64.409}\n",
    "2022-03-18 00:19:21,031 INFO    19:  {'compress_ratio': 62.36195696042758, 'flops': 38.36510432, 'params': 50053.078, 'accuracy': 0.3761879825312835, 'latency': 65.029}\n",
    "2022-03-18 00:19:21,031 INFO ------------------------------------------------\n",
    "```\n",
    "## 5. 部署推理服务（可选）\n",
    "如果需要对模型的性能进行调优，需要准备一台推理服务器（310或者710），在服务器上部署推理服务，详细步骤参考 https://support.huaweicloud.com/usermanual-mindstudio304/atlasms_02_0322.html AutoML工具->安装部署->启动推理服务  \n",
    "cann 安装流程及软件包获取参看https://www.hiascend.com/document/detail/zh/canncommercial/51RC1/envdeployment/instg \n",
    "mindspore 安装流程及部署参看https://www.mindspore.cn/install\n",
    "\n",
    "## 6. 分段模型调优配置介绍\n",
    "\n",
    "**general**部分主要配置如下：\n",
    "\n",
    "```yml\n",
    "general:\n",
    "  backend: mindspore  # 模型框架选择\n",
    "  parallel_search: True  # nas部分是否使用并行搜索\n",
    "  dataset_sink_mode: True\n",
    "  task:\n",
    "    local_base_path: ./tasks  # 开发者自行给出automl输出目录\n",
    "    task_id: \"yolov5_prune_parallel\"  # 开发者自行给出任务名称\n",
    "  device_evaluate_before_train: False #是否在训练之前进行310/710设备端评估\n",
    "  logger:\n",
    "    level: info  # logger等级可调为debug\n",
    "  worker:\n",
    "    timeout: 7200000\n",
    "```\n",
    "\n",
    "\n",
    "**pipline**部分配置如下：\n",
    "\n",
    "```yaml\n",
    "pipeline: [nas] # 量化调优只有nas过程\n",
    "```\n",
    "\n",
    "\n",
    "**nas**部分主要配置如下：\n",
    "\n",
    "```yaml\n",
    "nas:\n",
    "    pipe_step:\n",
    "        type: SearchPipeStep\n",
    "    model:\n",
    "        pretrained_model_file: \"/models/ssd_resnet50_fpn.ckpt\" # 预训练权重文件\n",
    "        model_desc:\n",
    "          type: PruneModel\n",
    "          model_file_path: /home/automl/ssd/eval.py # get_model函数所在文件\n",
    "          pkg_path: /home/automl/ssd/ # get_model函数所在文件夹路径\n",
    "        batch_size: 2\n",
    "        input_shape:\n",
    "          - type: fp32\n",
    "            tensor: True\n",
    "            shape: [ 1,3,640,640]\n",
    "\n",
    "    search_algorithm:\n",
    "        type: MsQuantRL\n",
    "        codec: QuantRLCodec\n",
    "        policy:\n",
    "            max_episode: 20   # 量化采样数量\n",
    "            num_warmup: 10    # time without training but only filling the replay memory, recommended:10-20\n",
    "        objective_keys: [ 'accuracy','compress_ratio','latency' ]\n",
    "        reward_type: 'acc_first'\n",
    "        custom_reward: False\n",
    "        latency_acc_ratio: 0.5   # ratio of latency to accuracy in reward.\n",
    "                             # If custom_reward is false,this value doesn't need to be configured.\n",
    "                             # Besides, if custom_reward is true,you can set latency_acc_ratio to 0 so that latency is not used in reward.\n",
    "                             # Otherwise,this value is recommended to be greater than 0.1.\n",
    "        stop_early: False\n",
    "        acc_threshold: 0.5\n",
    "        latency_threshold: 5\n",
    "        compress_threshold: 40\n",
    "\n",
    "\n",
    "    search_space:\n",
    "        type: SearchSpace\n",
    "        hyperparameters:\n",
    "            -   key: network.bit_candidates\n",
    "                type: CATEGORY\n",
    "                range: [ 8, 32]\n",
    "    trainer:\n",
    "      type: OriTrainer\n",
    "      seed: 234\n",
    "      callbacks: [MsQuantPTQCallback, CustomMetricCallback, CustomExportCallback]\n",
    "      calib_portion: 0.01\n",
    "      custom_calib: # 模型量化矫正函数配置\n",
    "        pkg_path: /home/automl/ssd/ # 矫正函数所在文件夹路径\n",
    "        path: /home/automl/ssd/train.py # 矫正函数所在文件\n",
    "        func: calib_func # 矫正函数名称\n",
    "      custom_eval: # 模型评估参数配置\n",
    "        pkg_path: /home/automl/ssd/ # 模型评估函数所在文件夹路径\n",
    "        path: /home/automl/ssd/eval.py # 模型评估函数所在文件\n",
    "        func: eval_net # 模型评估函数名称\n",
    "        metric_name: \"mAP\"\n",
    "\n",
    "    evaluator:\n",
    "        type: Evaluator\n",
    "        device_evaluator:\n",
    "          type: DeviceEvaluator\n",
    "          custom: QuantCustomEvaluator\n",
    "          backend: mindspore\n",
    "          om_input_shape: 'input_0:1,3,640,640'\n",
    "          hardware: \"Davinci\"\n",
    "          remote_host: \"http://x.x.x.x:xxxx\" # 310/710设备评估服务器地址\n",
    "          repeat_times: 1\n",
    "          muti_input: True\n",
    "          save_intermediate_file: True\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a01acdf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
